1. Field of the Invention
This invention relates to architectural frameworks for development of multimedia applications, and more specifically to architectural frameworks for developing adaptive, personalized, interactive multimedia applications and services.
2. Background and Material Information
In general, designing and implementing interactive systems is a complex and lengthy task. If one adds multimedia to the development equation, the level of complexity, the content variability and the required management support immediately soars and can overwhelm the development process. On the other hand, there presently exists a very dynamic and rich environment that potentially offers a business opportunity allowing one to build a family of applications that can be strongly differentiated by leveraging the same rich and complex content. Thus, a double edged sword exists.
If one examines the requirements, present and future, of information, more specifically multimedia information, one discovers that in general these requirements are a response to the xe2x80x9cdynamics of informationxe2x80x9d. These dynamics can be characterized by: constantly changing information; broad user population; and heterogenous landscape of delivery devices. If one grafts onto this picture the dynamics of collaboration or computer-supported work in synchronous or asynchronous mode, and potentially the technical problems are further compounded by the opportunity for differentiated and value-added services increases, i.e., the double-edged sword once again.
The best way to understand a system is to have an abstraction that describes a simpler picture of the structure and the machinery. A metaphoric vehicle is useful in that it allows framing of a problem and likewise offers a solution that supports and promotes flexibility, expressiveness, and scalability in information design and display. One can say that a multi-media presentation is like xe2x80x9ctelling a storyxe2x80x9d. The presentation author is attempting to convey a communicative intent and more than likely it was constructed with a particular audience in mind, as well as a specific context and medium.
The computational narrative model, as disclosed in Brooks, K. M., xe2x80x9cDo Agent Stories Use Rocking Chairs: The Theory and Implementation of One Model for Computational Narrativexe2x80x9d, Processings of the Fourth ACM International Multimedia Conference on Intelligent User Interfaces, ACM Press 1996 and Murtaugh, M. xe2x80x9cThe Automatist Storytelling System: Putting the Editor""s Knowledge in Softwarexe2x80x9d, MIT MS Thesis, 1996, offers a metaphor for creating tools that are capable of going beyond traditional storytelling by enhancing the editorial through the leveraging of the computer""s ability to support rapid decision making. According to Brooks, narrative represents the universe of story elements for a given story, i.e., the collection of possibility, and narration as a specific navigation through that universe.
As shown in FIG. 1, the process of computational storytelling involves the author supplying the elements of the story and the structure to organize the story elements. The agent takes the elements of the story and the structure and generates a story, more precisely, a narrative, and presents the xe2x80x9cstoryxe2x80x9d to an audience. The audience reacts and generates feedback to the agent. The agent acting as proxy for the author can react to the feedback by modifying the presentation.
Some current conceptual views regarding the techniques or technical strategies that are related to developing a framework for creating and delivering interactive multimedia applications include: dynamic presentation, behavior-based artificial intelligence, memory-based learning, and user modeling.
Regarding dynamic presentation, Maybury, M. xe2x80x9cIntelligent Multimedia Interfacesxe2x80x9d, AAAI/MIT Press, Cambridge, Mass., 1993, discloses that automatic multimedia presentation involves the stages of content selection (i.e., what to say), media allocation (i.e., what media to present it in), and media realization (i.e., how to say it). The focus is the media allocation and realization phase. More specifically, how to create presentations without knowing all xe2x80x9cfactsxe2x80x9d during design time. The basic objective is to enable the creation of user interfaces that are sufficiently flexible and adaptive to xe2x80x9cre-inventxe2x80x9d themselves at run-time. To support this flexibility and adaptability, an interface needs to be developed not to a final fixed form, but to some protean form that can be reshaped at run time, time after time, to meet the requirements of any situation that invalidates its current form.
Szekely P., xe2x80x9cRetrospective and Challenges for Model-Based Interface Developmentxe2x80x9d, USC Information Sciences Institute, Marina del Rey, Calif., 1996, proposes one architecture. Szekely discloses that a model-based user interface calls for a model of the interface that is organized as three levels of abstraction: task and domain model for the application, an abstract user interface specification, and a concrete user interface specification. The task model represents the task that the user will undertake to perform with the application. The domain model represents the data and the operations that are part of an application.
The second level, according to Szekely, is the abstract user interface specification. At this level, an interface is defined in terms of abstract interaction units, information elements, and presentation units. The abstract interaction units are low-level interactions such as showing a presentation unit. Information elements represent data such as attributes extracted from the domain model. Presentation units are abstractions of windows and specify collections of abstract presentation units and information elements that are to be treated as a unit. Basically, the abstract user interface specification abstractly specifies the way information will be presented in the interface and form for interaction with the information.
The third level, according to Szekely, is the concrete user interface specification that specifies rendering styles for the presentation units, i.e., widgets. Different model-based user interface (UI) frameworks differ in what models they provide. Szekely discloses that some frameworks have one model but not the other two, while in other cases, only one model is defined. FIG. 2 is a flowchart showing a generic model-based presentation system as disclosed in Szekely.
An alternative reasoning framework has emerged in Artificial Intelligence circles called Behavior-Based AI (BBAI) as disclosed in Maes, P. xe2x80x9cBehavior-Based Artificial Intelligencexe2x80x9d, Proceedings of Second Animat Conference on Adaptive Behavior, 1992. This new approach represents more of a different way of thinking about a problem domain than an alternative reasoning technique. The knowledge-based approach involves capturing the rules to solve a domain. In contrast, the BBAI approach relies on a set of lower level competencies which are each experts at solving one part of the larger problem domain as disclosed in Brooks.
Additionally, the BBAI approach tends to emphasize the system behavior as opposed to the system knowledge. Furthermore, BBAI stresses that the system should be situated in its environment and have direct (or as close as possible) access to the problem domain. This framework enables a system to bring together different classes of reasoning techniques, heuristic, statistical, etc., and incorporate each application of a technique into a lower-level competency module or xe2x80x9cexpertxe2x80x9d. In effect, these modules come together to form a multi-agent system.
Another learning technique, as disclosed in Stanfield, C. et al., xe2x80x9cToward Memory-Based Reasoningxe2x80x9d, Communications of the ACM, 20(12), ACM Press, 1986, is memory-based learning. Basically, memory-based learning entails comparing a new situation against each of the situations which have occurred before. Given a new situation, a memory-based learning agent looks at the actions taken in N of the xe2x80x9cclosestxe2x80x9d situations or xe2x80x9cnearest neighborsxe2x80x9d to predict the action for a new situation. FIG. 3 shows a diagram of the memory-based reasoning approach.
User modeling is an inexact science but its predictions need not be perfect to be useful. User models can range from simply storing a bit indicating if the user is a novice or expert in terms of an application, to a rich, complex snapshot of the user""s interest and preferences. Once a universe of user models is collected and maintained, the models may serve as data for further analysis to find pattern and trends in this universe. These are some of many critical issues relevant to user modeling.
User models may be either pragmatic or cognitive as disclosed in Orwant, J, xe2x80x9cDoppelganger Goes To School: Machine Learning for User Modeling,xe2x80x9d, MIT MS Thesis, 1993. The cognitive type user models are not connected to any application or applications in particular. This type of user model is attempting to capture a user""s beliefs, goals and plans in a general sense. A pragmatic user model is not driven by a cognitive model but by the practical aspects of the environment, e.g., applications. The pragmatic user model can be characterized by the collection of raw observational data and making sense of the data after the fact. In another sense, the cognitive model is a top down approach and the pragmatic model is a bottom up approach.
Conceptually, individuals can take on particular roles, e.g., business, leisure, parental, professional. These are defined as persona in a user modeling sense. Personae could be utilized to partition the user model space into more manageable chunks.
A pragmatic user model can make use of filtering techniques. Content-based filtering involves selecting items for the user based on correlations between content of the items and the user""s preferences. For example, a personalized TV program guide uses information about a television program, such as the program""s type and its level of violence to predict whether or not to recommend and include the show in a personalized line-up. Generally, users rely on exploration to discover new items of interest, i.e., serendipitous items. By definition, content-based filtering has no inherent capability to generate these sort of items. In practice, one must add special purpose techniques to add these capabilities to content-based filtering to introduce serendipity. For example, a user might be unaware of their interest in true crime shows until she actually comes across xe2x80x9cAmerica""s Most Wantedxe2x80x9d. Assuming no indications of this trend had previously surfaced, content-based filtering would have never detected this particular interest. Content-based filtering simply does not allow a user to expand their interests.
Social-based filtering is one potential solution to the serendipity dilemma. Social-based filtering basically attempts to exploit similarities between the profiles of different users to filter content. Social-based filtering can be an extension of content-based filtering. Once a user model is constructed and is being maintained, social-based filtering algorithms can compare this model to other user models and weigh each model for the level of similarity with the user model. Orwant, J., xe2x80x9cFor Want of a Bit The User Was Lost: Cheap User Modelingxe2x80x9d, IBM Systems Journal, vol. 35, Nos 3and4, 1996 and Shardanand, U., xe2x80x9cSocial Information Filtering for Music Recommendationxe2x80x9d, MIT MS Thesis, 1994 disclose algorithms for computing similarity between user models.
Accordingly, the present invention is directed to a method for design of an adaptive personalized interactive content delivery system that substantially obviates one or more of the problems arising from the limitations and disadvantages of the related art.
It is an object of the present invention to provide an architectural framework that is composed of a collection of classes for building interactive multimedia applications and services.
It is a further object of the present invention to provide an architectural framework that will enable a developer to build up locations that deliver services that dynamically adapt to the user, the content, and the delivery context, resulting in an effective contextual personalized on-line experience.
Another object of the present invention is to provide an architectural framework that supports and promotes the creation of reusable components for building personalized interactive multimedia presentations of complex applications.
Accordingly, one aspect of the present invention is directed to a method for creating and delivering an interactive multimedia application that can dynamically adapt to at least one user. At least one user model is created for at least one user, the at least one user model represents interests and trends of the at least one user. A multimedia story is developed based on the at least one user model. A customized presentation of the multimedia story is generated where the at least one multimedia story allows for multiple presentations of the multimedia story. The customized presentation is displayed to the at least one user. The customized presentation is modified based on input from the at least one user.
In another aspect of the present invention, the story includes a protean-like narrative.
In still another aspect of the present invention, the creating includes: gathering data from the at least one user; analyzing a history of the at least one user; monitoring data related to the at least one user; detecting patterns and trends of the at least one user; and preparing the at least one user model based on the gathering, analyzing, monitoring, and detecting. The at least one user model is modified periodically based on information obtained from periodically repeating the gathering, analyzing, monitoring, and detecting.
In a further aspect of the present invention, the at least one user model includes a set of models.
In another aspect of the present invention, the story includes at least one content element. The at least one content element characterizes data of the interactive multimedia application. The at least one content element is representable in multiple forms.
In still another aspect of the present invention, the at least one user model comprises a set of models.
In a further aspect of the present invention, the multiple forms include text, audio, video, image, or multimedia.
In another aspect of the present invention, the invention includes filtering the at least one content element to produce a subset of the at least one content element, each content element in the subset of at least one content elements selected based on semantics of the filtering.
In still another aspect of the present invention, the invention includes assembling the subset of at least one content elements to produce the multimedia story. The multimedia story may be personalized to the at least one user.
In a further aspect of the present invention, the generating includes: determining the delivery environment of the at least one user; determining the style look and feel for the presentation; determining the narrative context for the presentation. The narrative context defined by the semantics of the interactive multimedia application; and creating a customized presentation of the multimedia story based on the delivery environment, the style look and feel, and the narrative context.
In another aspect of the present invention, a weighted value may be assigned to each interest and trend of the at least one user. The weighted value represents the relative importance of each interest and trend with respect to the at least one user""s apparent interests.
In still another aspect of the present invention, the interactive multimedia application may be created using object-oriented design techniques.
In a further aspect of the present invention, the invention is directed to a method for creating and delivering an interactive multimedia application that can dynamically adapt to at least one user that includes: creating a story engine, the story engine may be created by the interactive multimedia application; creating a user model manager, the user model manager may be created by the interactive multimedia application; providing the story engine with application-specific information and user information; providing the story engine with a user model from the user model manager, the user model represents interests and trends of the at least one user; providing the story engine with a narrative structure, the narrative structure defined by the semantics of the interactive multimedia application; producing user-relevant content, the user-related content may be produced by applying filters to the content model, the user model may be used for filtering purposes; creating a presentation engine, the presentation engine may be created by the interactive multimedia application; providing the presentation engine with the narrative structure, content model, and a presentation model, the content model may be empty; generating an abstract presentation defined by the presentation model, the abstract presentation may be generated by the presentation engine; generating a concrete presentation by using the abstract presentation""s heuristics, the concrete presentation may be generated by the presentation engine; and displaying the concrete presentation by the presentation engine, wherein the abstract presentation and the presentation engine autonomously handle interaction scenarios, and trends and patterns are periodically recomputed based on interaction histories and the user models, the interactive multimedia application may be self-improving and self-sustaining.
In another aspect of the present invention, the interactive multimedia application may be created using object-oriented design techniques.
In still another aspect of the present invention, the interactive multimedia application may be created using JAVA.
In a further aspect of the present invention, the invention is directed to a system for creating and delivering interactive multimedia applications that dynamically adapt to a user that include: a user modeling subsystem where the user modeling subsystem creates and maintains at least one user model for each user, each at least one user model represents interests and trends of each user; a story engine subsystem where the story engine subsystem selects appropriate content elements and collects and organizes these elements in accordance with a narrative framework; and a presentation subsystem where the presentation subsystem generates a presentation to the user, the presentation generated uses the narrative framework.
In another aspect of the present invention, the user modeling subsystem includes: a user model editor; a user modeling manager; an analysis engine; and a user model database.
In still another aspect of the present invention, the story engine subsystem includes: a first database where the first database contains a content model library, the first database accesses content from a content database; and a second database where the second database contains a story template library.
In a further aspect of the present invention, the presentation subsystem includes: a first database where the first database contains at least one presentation models; a presentation builder; a second database where the second database contains a concrete presentation library; and a presentation engine.
In another aspect of the present invention, the content elements may represent pieces of information that can be presented via one or more media types.
In still another aspect of the present invention, the presentation may be constrained by a narrative style, narrative context, and demands of the delivery environment of the user.